Learning Hierarchical Structures with Linear Relational Embedding

We present Linear Relational Embedding (LRE), a new method of learning a distributed representation of concepts from data consisting of instances of relations between given concepts. Its final goal is to be able to generalize, i.e. infer new instances of these relations among the concepts. On a task involving family relationships we show that LRE can generalize better than any previously published method. We then show how LRE can be used effectively to find compact distributed representations for variable-sized recursive data structures, such as trees and lists. 1 Linear Relational Embedding Our aim is to take a large set of facts about a domain expressed as tuples of arbitrary symbols in a simple and rigid syntactic format and to be able to infer other “common-sense” facts without having any prior knowledge about the domain. Let us imagine a situation in which we have a set of concepts and a set of relations among these concepts, and that our data consists of few instances of these relations that hold among the concepts. We want to be able to infer other instances of these relations. For example, if the concepts are the people in a certain family, the relations are kinship relations, and we are given the facts ”Alberto has-father Pietro” and ”Pietro has-brother Giovanni”, we would like to be able to infer ”Alberto has-uncle Giovanni”. Our approach is to learn appropriate distributed representations of the entities in the data, and then exploit the generalization properties of the distributed representations [2] to make the inferences. In this paper we present a method, which we have called Linear Relational Embedding (LRE), which learns a distributed representation for the concepts by embedding them in a space where the relations between concepts are linear transformations of their distributed representations. Let us consider the case in which all the relations are binary, i.e. involve two concepts. In this case our data consists of triplets (concept1; relation; concept2), and the problem we are trying to solve is to infer missing triplets when we are given only few of them. Inferring a triplet is equivalent to being able to complete it, that is to come up with one of its elements, given the other two. Here we shall always try to complete the third element of the triplets 1. LRE will then represent each concept in the data as a learned vector in a Methods analogous to the ones presented here that can be used to complete any element of a triplet can be found in [4]. Euclidean space and each relationship between the two concepts as a learned matrix that maps the first concept into an approximation to the second concept. Let us assume that our data consists of C such triplets containing N distinct concepts and M binary relations. We shall call this set of triplets C; V = fv1; : : : ;vNg will denote the set of n-dimensional vectors corresponding to the N concepts, and R = fR1; : : : ; RMg the set of (n n) matrices corresponding to the M relations. Often we shall need to indicate the vectors and the matrix which correspond to the concepts and the relation in a certain triplet c. In this case we shall denote the vector corresponding to the first concept with a, the vector corresponding to the second concept with b and the matrix corresponding to the relation with R. We shall therefore write the triplet c as (a; R;b) where a;b 2 V andR 2 R. The operation that relates a pair (a; R) to a vector b is the matrix-vector multiplication, R a, which produces an approximation to b. If for every triplet (a; R;b) we think of R a as a noisy version of one of the concept vectors, then one way to learn an embedding is to maximize the probability that it is a noisy version of the correct completion, b . We imagine that a concept has an average location in the space, but that each “observation” of the concept is a noisy realization of this average location. Assuming spherical Gaussian noise with a variance of 1=2 on each dimension, the discriminative goodness function that corresponds to the log probability of getting the right completion, summed over all training triplets is: